Mmcrl&oo 


1 - 

% 

\ . ^ L.  \ , . •%..  r-'V? 

Clearinghouse  Evaluators'  Guide 

for 

Microcomputer  - Based  Courseware 


Curriculum 


EVALUATOR'S  GUIDE 
FOR 

MICROCOMPUTER-BASED 

COURSEWARE 


CANADA  N AT  ION  Ar*” LIBRARY  I 
B;6LiOiHE<5UE  NATIONALS  I 


DEVELOPED  BY 

THE  CLEARINGHOUSE 

OF 

ALBERTA  EDUCATION 


1 1 160  Jasper  Avenue 
Edmonton,  Alberta 
T5K  0L2 


Digitized  by  the  Internet  Archive 
in  2016 


https://archive.org/details/evaluatorsguidefOOalbe 


TABLE  OF  CONTENTS 


paKe 

l.  INTRODUCTION 

A.  PURPOSE  1 

B.  PROCESS  1 

C.  DESCRIPTION  OF  THE  INSTRUMENT  2 

II.  PROGRAM  EVALUATION  FORM 

A.  PEDAGOGICAL  CONTENT  3 

B.  INSTRUCTIONAL  FORMAT  6 

C.  TECHNICAL  CHARACTERISTICS  1 4 

D.  OBSERVATION  OF  STUDENT  USERS  16 

m.  DISK  EVALUATION  FORM 

A.  DESCRIPTION  17 

B.  IMPLEMENTATIONAL  SUPPORT  17 

C.  OBJECTIVES  20 

D.  DISK  EVALUATION  SUMMARIES  21 

E.  SUMMATION  22 

F.  SUGGESTIONS  TO  THE  PRODUCER  22 

G.  RECOMMENDATION  22 

IV.  GUIDELINES  FOR  MAKING  RECOMMENDATIONS  23 

V.  DEFINITIONS  25 

VI.  GLOSSARY  OF  TERMS  26 


' 


■ 


I.  INTRODUCTION 


A.  PURPOSE 

Except  in  instructional  situations  where  student  programming  is  the  primary 
activity,  microcomputers  can  be  used  to  assist  the  instructional  process 
effectively  only  to  the  extent  that  good  quality  courseware  is  available. 
This  Evaluator's  Guide  and  the  associated  evaluation  forms  have  been 
designed  to  assist  teachers  in  identifying  such  courseware. 

The  development  of  the  guide  and  the  evaluation  instruments  has  been  based 
on  the  philosophical  approach  that  description  and  evaluation  of  courseware 
should  be  separated.  This  has  been  done  in  the  four  major  sections  of  the 
instrument  - PEDAGOGICAL  CONTENT,  INSTRUCTIONAL  FORMAT, 
TECHNICAL  CHARACTERISTICS,  and  I MPLE MENTATION AL  SUPPORT. 


B.  PROCESS 

The  process  of  evaluation  conducted  by  Alberta  Education  involves  officials 
within  the  Program  Development  Division.  As  priorities  are  identified,  the 
Clearinghouse  solicits  courseware  from  publishers  for  evaluation.  Materials 
which  pass  an  initial  screen  are  described  and  assessed  by  a minimum  of 
three  different  teachers  who  have  the  appropriate  knowledge  in  the  subject 
area  under  consideration  and  who  have  received  training  in  the  use  of  the 
evaluation  instrument. 

Courseware  favourably  reviewed  by  the  teachers  is  evaluated  further  by  the 
Program  Development  Division  for  congruency  to  the  Programs  of  Studies 
and  may  be  designated  as  a "Prescribed",  "Recommended"  or 
"Supplementary"  learning  resource  for  Alberta  schools.  Prescribed  and 
Recommended  materials  are  acquired  and  distributed  through  the  School 
Book  Branch  of  Alberta  Education.  All  courseware  evaluations  are  filed  in 
the  Clearinghouse  and  results  are  made  available  to  Alberta  schools. 
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C.  DESCRIPTION  OF  THE  INSTRUMENT 

Courseware  currently  being  published  may  come  in  a number  of 
configurations. 

a.  A disk  may  contain  a number  of  different  ’’stand-alone"  programs. 
For  example,  programs  on  spelling,  identification  of  capital  cities 
and  review  of  basic  number  facts  may  be  available  all  on  one  disk. 

b.  A disk  may  contain  a number  of  programs  tied  together  under  the 
same  basic  goals  but  each  having  different  objectives.  For 
example,  a disk  may  have  four  programs  - addition,  subtraction, 
multiplication  and  division  of  decimals. 

c.  A disk  may  contain  one  main  program,  for  example,  a simulation  of 
a scientific  experiment. 

d.  Two  or  more  disks  may  be  tied  together  into  a package  or  series. 
For  example,  six  disks  reviewing  arithmetic  fundamentals  may  be 
combined  into  a single  package  with  a management  system. 

As  the  basic  item  of  purchase  is  the  disk,  IT  IS  THE  DISK  which  has  been 
made  the  basis  of  evaluation  of  this  instrument. 

There  are  two  parts  to  the  evaluation  instrument  - the  PROGRAM 
EVALUATION  FORM  and  the  DISK  EVALUATION  FORM. 

A PROGRAM  EVALUATION  FORM  is  generally  completed  for  each  stand- 
alone program  on  the  disk.  The  form  has  descriptions  and  evaluations  of  the 
program's  pedagogical  content,  instructional  format  and  technical 
characteristics.  Each  stand-alone  program  on  the  disk  may  receive  an 
evaluation  that  is  independent  of  other  programs  on  the  disk. 

A disk  containing  a number  of  programs  tied  together  under  a common  goal 
will  generally  have  each  of  the  programs  evaluated  separately.  However, 
when  the  programs  are  designed  in  such  a manner  that  the  only  noticeable 
difference  between  them  is  in  the  pedagogical  content  and  all  other  factors 
of  instructional  format  and  technical  characteristics  are  identical,  one 
PROGRAM  EVALUATION  FORM  may  be  used  for  all  the  programs. 

A DISK  EVALUATION  FORM  is  used  to  evaluate  the  overall  DISK.  It 
contains  the  identification  of  the  disk  (title,  publisher,  etc.),  and  a 
description  and  evaluation  of  the  implementational  support  that  is  provided. 
It  also  contains  observations  of  students,  objectives,  summative  evaluation 
statements,  suggestions  to  the  producer  and  a final  recommendation. 
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n.  PROGRAM  EVALUATION  FORM 
A.  PEDAGOGICAL  CONTENT 

The  pedagogical  content  of  the  disk  includes  an  outline  of  the  content  range, 
sequence  and  depth  along  with  other  aspects  of  content  such  as  accuracy 
and  readability.  Pedagogical  content  is  the  first  thing  that  evaluators 
should  look  at  - if  the  content  is  not  appropriately  presented,  all  the 
powerful  potential  of  the  computer  is  negated. 

Programs  with  good  content  RANGE  present  material  that  is  appropriate  for 
the  target  audience  to  cover  in  one  sitting.  The  program  should  also  begin 
and  end  at  a point  that  is  appropriate  for  the  skills/abilities  of  the  intended 
users.  An  example  of  poor  range  for  a primary  class  might  be  a program 
which  starts  with  two  digit  by  one  digit  multiplication  but  does  not  include 
any  coverage  of  the  content  before  that.  Another  example  of  poor  range 
might  be  a program  that  attempts  to  teach  the  solution  of  linear  equations 
from  "2X  = 4"  up  to  "3(X  - 6)  = 5(4  - 2X)  - (X  - 6)"  in  one  sitting. 

Programs  which  have  been  designed  to  instruct  and/or  to  provide  practice 
should  have  a content  SEQUENCE  that  will  enable  the  user  to  learn  the 
skill/concept.  Common  faults  include  the  excessive  size  of  incremental 
steps  and  the  absence  of  an  easy  to  difficult  sequence.  An  example  of  poor 
sequence  might  be  the  program  which  takes  the  student  from  an 
introduction  to  fractions  to  addition  of  fractions  without  any  work  on 
equivalence.  A drill  designed  to  review  basic  addition  facts  might  be 
considered  to  have  poor  sequence  if  ail  students  were  required  to  work 
through  a sequence  of  10  levels,  each  with  10  questions  before  they  arrived 
at  the  questions  that  were  appropriate  to  their  grade  level.  It  should  be 
noted  that  not  all  programs  require  a sequential  development  and  evaluators 
will  have  to  base  their  assessment  on  the  intents  of  the  program. 

DEPTH  refers  to  the  intensity  with  which  content  is  presented  or  practiced 
and  its  effectiveness  must  be  measured  against  user  needs  and  program 
intents.  Poor  depth  such  as  insufficient  instruction  and/or  insufficient 
practice  is  a common,  major  weakness.  An  example  of  poor  depth  would  be 
requiring  users,  new  to  a skill,  to  do  only  two  exercises  before  being 
advanced  to  the  next  skill  level.  Other  examples  of  poor  depth  are  showing 
only  one  example  of  an  operation  or  providing  only  one  screen  page  of 
explanation  when  classroom  experience  would  indicate  that  extensive 
instruction/practice  is  necessary. 

While  CONTENT  ACCURACY  is  usually  satisfactory,  programs  will 
occasionally  have  an  error,  e.g.  in  a mathematical  or  scientific  formula.  It 
is  more  common  to  find  inaccurate  simulations  such  as  one  based  on  a model 
that  has  been  so  simplified  as  to  make  it  misleading. 

Courseware  with  poor  READABILITY/CLARITY  would  typically  have 
vocabulary  beyond  the  target  audience’s  reading  level,  overly  complicated 
sentence  structure,  unduly  long  sentences  or  excessive  quantity  of  text. 
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PEDAGOGICAL 

CONTENT 

DESCRIPTION 

Program 

Contents 


Write  narrative  statements  which  will  describe  the 
following  aspects  of  the  content. 


Describe  the  pedagogical  content  of  the  program 
(topics,  skills,  concepts). 

1.  What  are  the  topics/skills/concepts  covered  in  the 
program?  (range) 

2.  What  is  the  progression  of  the  presentation?  How 
are  the  units  ordered?  (sequence) 

3.  Describe  the  depth  of  treatment.  What  does  the 
treatment  include?  (explanations,  rules,  examples, 
illustrations,  drills,  tests,  etc.) 

Example:  This  program  covers  three  types  of  addition 
of  fractions.  Students  begin  with  addition  of  common 
fractions  with  the  same  denominators,  proceed  to 
addition  of  fractions  where  one  fraction  has  a 
denominator  that  is  a factor  of  the  other  (e.g.  2 and  6, 
3 and  12)  and  finish  the  program  with  addition  of 
fractions  with  completely  different  denominators  (e.g. 
3 and  4 or  5 and  7).  Improper  and  mixed  fractions  are 
not  included  in  the  program.  The  presentation  of  each 
topic  includes  one  page  of  introduction,  two  examples 
and  a brief  drill.  A statement  of  rules  is  available.  The 
program  ends  with  a test  on  all  three  skills. 

Example:  The  program  covers  four  job  topics  -attitude, 
work  habits,  task  performance  and  interpersonal 
relationships.  Each  is  presented  in  two  or  three  pages 
of  text  followed  by  5 multiple  choice  questions.  A 
true/false  10  item  test  checks  student  comprehension. 

Example:  The  program  covers  the  different  parts  of 

the  newspaper  and  the  role  that  each  plays  in 
communicating  to  the  public.  The  user  selects  from  six 
choices  at  the  menu  (local  news,  editorial  page,  sports, 
leisure,  business  and  ads).  Each  lesson  contains  an 
explanation  of  the  purpose  of  the  section,  sample  pages 
and  reference  to  support  materials.  Multiple  choice 
questions  also  test  the  student's  understanding  of  the 
content.  After  the  student  has  shown  mastery  of  all  six 
programs,  an  optional  10  item  drill  in  game  format 
provides  practice  in  identifying  the  appropriate  location 
of  various  articles. 
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PEDAGOGICAL 

CONTENT 

EVALUATION 


Range 


Sequence 


Depth 


Accuracy 


Write  narrative  statements  which  will  evaluate  the 
content  described  earlier.  The  following  questions  are 
provided  to  assist  you  in  supplying  evaluative 
comments. 

1.  Is  the  range  of  the  content  appropriate  for  the 
target  audience?  Is  the  student's  attention  span 
exhausted  by  the  time  a "unit"  is  completed? 

2.  Is  the  range  of  the  content  effective  in  achieving 
the  intents  of  the  program?  Is  sufficient  content 
covered?  Is  too  much  covered? 

3.  Is  the  sequence  of  presentation  appropriate  for  the 
target  audience? 

- Is  the  size  of  the  increment  appropriate? 

- Is  the  content  sequence  effectively  and 
appropriately  designed  (e.g.  easy  to  difficult, 
concrete  to  abstract,  random  mix)? 

4.  Is  the  sequence  effective  in  achieving  the  intents 
of  the  program? 

3.  Where  appropriate,  can  the  -user  enter  into  the 
sequence  at  intermediate  steps  or  must  (s)he 
always  start  at  the  beginning? 

6.  Does  the  sequence  remain  consistent/appropriate 
during  and  after  branching? 

7.  Is  the  depth  appropriate  for  the  target  audience? 

8.  Is  the  depth  effective  in  achieving  the  intents  of 
the  program? 

- Are  definitions  and  explanations  available  where 
necessary? 

- Are  definitions  and  explanations  effective? 

- Does  the  instruction  have  an  appropriate  amount 
of  detail  to  be  effective? 

- Is  there  a sufficient  number  of  examples, 
illustrations,  exercises,  etc.? 

- Is  the  user  given  an  appropriate  number  of 
opportunities  to  use  the  concept? 

- Is  the  concept/skill  presented  an  appropriate 
number  of  times? 

9.  Is  the  content  accurate?  Does  it  contain  factual 
errors,  an  invalid  model  in  a simulation,  an 
oversimplified  model  or  series  of  examples, 
inaccurate  displays? 


10.  Is  the  content  current? 
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Bias  11.  Is  the  content  balanced  or  biased? 

- e.g.  social  class/social  role  reference? 

- e.g.  ethnic/racial  references? 

- e.g.  reference  to  sexual  roles? 

- e.g.  references  to  violence? 

- e.g.  age  portrayals? 

- e.g.  stereotyping? 

Are  there  unfair  or  inaccurate  judgements? 

Does  it  nurture  the  development  of  a positive 
self-concept? 

Does  it  contain  Canadian  data  where  relevant? 

Is  the  content  metricated  where  appropriate? 

Readability  12.  Are  vocabulary,  sentence  structure,  sentence 
length,  quantity  of  text  and  general  readability 
appropriate  for  the  target  audience? 


13.  Is  the  content  presented  in  a clear  and  concise 
manner  which  is  appropriate  for  the  target 
audience? 

Relevance  14.  Are  interest  levels,  visual  images  and  levels  of 
abstraction  relevant  for  the  target  audience? 

Media  15.  Is  the  content  appropriate  to  the  medium?  Is  it 

best  taught  on  a computer  or  through  some  other 
medium?  Should  other  media  be  used  in 
conjunction  with  the  computer? 


B.  INSTRUCTIONAL  FORMAT 

Instructional  Format  sections  focus  on  the  utilization  of  the  capabilities  of 
the  computer  in  presenting  the  pedagogical  content  described  and  evaluated 
earlier.  These  sections  cover  such  features  as  interaction,  questioning 
technique,  feedback,  evaluation,  branching  and  methods  of  control. 

If  the  capabilities  of  the  computer  to  individualize  instruction  are  to  be  used 
fully,  the  student  must  be  given  an  opportunity  to  interact  with  the  program 
and  (s)he  must  find  this  INTERACTION  reasonably  easy  to  accomplish. 
Evaluating  the  effectiveness  of  the  interaction  is  partly  an  inquiry  into 
whether  the  user’s  attention  will  be  distracted  from  his/her  study  of  the 
pedagogical  content  that  is  being  presented  by  an  unnecessarily  complicated 
mode  of  interaction  and  partly  an  assessment  of  the  value  of  that 
interaction  in  promoting  the  learning.  Poor  interaction  is  most  commonly 
found  in  tutorials,  where  the  user  tends  to  turn  pages  only,  and  in 
'educational  games'  where  the  means  of  input  often  can  be  overly 
complicated.  Examples  of  poor  interaction  may  be  found  in  programs  which: 
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- have  unduly  complicated  entry  methods  which  emphasize  keyboard 
manipulation  skills  to  the  detriment  of  the  content  skills  being 
presented; 

- have  inconsistent  input  methods; 

- have  the  user  frequently  pressing  the  space  bar  rather  than 
interacting  with  the  program  in  a meaningful  way; 

- have  the  user  only  passively  involved  in  the  program. 

If  a computer's  full  capabilities  are  to  be  effectively  utilized,  tutorials 
and  drills  must  advance  beyond  the  typical  textbook/drill  sheet 
approach.  QUESTIONING  TECHNIQUES  that  can  enhance  educational 
software  are: 

- Where  appropriate,  drill  questions  should  be  posed  in  an  unpredictable 
pattern.  Additionally,  if  the  drill  is  used  again,  the  order  of  the 
questions  should  be  different. 

- Questions  with  which  the  user  has  had  trouble  can  be  stored  and  re- 
presented to  the  student  during  the  drill  or  upon  its  completion. 

- It  is  not  unusual  in  many  programs  to  find  the  same  question 
reappearing  immediately  or  shortly  after  it  was  successfully 
answered.  The  program  should  ensure  that  this  does  not  happen. 

The  design  of  appropriate  FEEDBACK  techniques  is  critical  in  making 
full  use  of  the  computer's  capabilities  but  unfortunately  most  of  the 
feedback  procedures  commonly  used  consist  solely  of  the  provision  of 
the  correct  answer.  As  a result,  many  programs  provide  nothing  more 
than  what  is  available  in  a textbook  with  the  answers  in  the  back. 

a.  In  evaluating  the  user's  input,  the  program  should  ensure  that  a 
correct  judgement  has  been  made.  This  includes  not  only  ensuring 
that  the  correct  answer  is  accepted  but  also  that  slight  differences 
in  format  are  also  accepted  (with  or  without  a cautionary  note  to 
the  student).  For  example: 

i.  a program  should  not  reject  "0.5"  in  favour  of  ".5"; 

ii.  a program  should  not  reject  "50  %"  in  favour  of  "50%"  (inserted 
space); 

iii.  "cat"  should  not  be  rejected  for  " cat"  or  "cat  " (leading  or 
trailing  space). 

b.  Feedback  should  be  motivational  and  sensitive  to  the  user's  needs. 
Significant  advantages  of  CAI  are  the  infinite  patience  and  the 
privacy  with  which  interaction  occurs.  These  are  negated  by: 

i.  a comment  such  as  "You  aren't  doing  very  well;  call  your 
teacher"; 

ii.  a comment  such  as  "You  haven't  put  much  thought  into  this,  have 
you?" 

iii.  a comment  such  as  "Error!  Error!  Error!" 

iv.  loud  'raspberries'  for  incorrect  answers; 

v.  unduly  time  consuming,  repetitive,  positive  graphic  rewards. 

c.  Cues  and  prompts  can  be  given  after  a wrong  answer  to  assist  the 
student  in  understanding  his/her  error.  A comment  such  as  "Did  you 
forget  to  move  the  decimal?"  is  much  more  effective  than  "Wrong". 
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d.  If  a computer  program  is  to  be  more  than  a textbook  with  answers  in 
the  back,  the  user  has  to  be  given  some  help  when  an  error  is  made. 
Corrective  feedback  showing  the  steps  in  the  solution  or  re- 
explaining the  concept  is  more  helpful  than  merely  telling  the 
student  tne  correct  answer  was  "present  participle". 

e.  Negative  feedback  should  not  be  more  attractive  than  the  positive, 
e.g.  a student  may  purposely  enter  incorrect  answers  in  order  to  see 
a boulder  fall  into  a pit  and  "squash"  a graphical  figure. 

f.  Summary  feedback  should  be  provided  after  the  user  has  completed 
the  drill  or  tutorial. 

The  inclusion  of  EVALUATION  TECHNIQUES  such  as  pre-tests  and 
post-tests  within  a program  can  enhance  its  potential  use  considerably. 
Having  students  take  a test  before  starting  on  some  educational 
software  would  allow  the  teacher  to  filter  out  those  students  who  are 
not  yet  ready  for  the  content  or  those  who  have  already  mastered  it. 
Post-tests  are  equally  important  as  their  results  will  indicate  who  did  or 
did  not  master  the  content.  Poorly  designed  pre-tests/post-tests  would 
have  inappropriate  criteria  and/or  criteria  that  can  not  be  changed  by 
the  teacher  or  questions  that  do  not  reflect  the  intents  and  content  of 
the  program. 

The  computer’s  effectiveness  in  individualizing  instruction  is  greatly 
enhanced  through  the  inclusion  of  BRANCHES  within  a program.  An 
effective  branch  would  provide  for  individual  needs  by  presenting 
suitable  remedial/enrichment  content.  Entry  into  and  exit  out  of  the 
branch  should  be  suitably  controlled  by  the  user  or  the  teacher.  For 
example,  automatic  entry  into  a remedial  branch  after  2 errors  might 
be  disadvantageous  for  a student  who  made  input  errors. 

The  computer's  effectiveness  in  individualizing  instruction  can  be 
enhanced  by  giving  the  student/teacher  appropriate  CONTROL  over 
some  program  features.  It  is  generally  expected  that  the  student  should 
have  control  over  the  pace  of  the  presentation  - this  is  usually  achieved 
by  pressing  the  space  bar  or  return  key.  The  appropriateness  of  other 
types  of  control  will  vary  according  to  the  age  of  the  target  audience 
and  the  ease  with  which  control  is  implemented. 


INSTRUCTIONAL 

FORMAT 

DESCRIPTION 


Write  narrative  statements  which  will  describe  the 
following  aspects  of  instructional  format. 


Student 

Interaction 


Describe  the  student  interaction  that  is  required  in  the 
program. 


1. 


What  types  of  student  interaction  are  there?  (multiple 
choice  entry,  input  of  answer,  "press  return",  timed 
responses,  calculator  function,  ability  to  work  a 
question  out  on  the  screen,  etc.) 
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Questioning 
T echniques 


Feedback 

Techniques 


2.  Comment  on  the  frequency  of  interaction. 

Example:  Student  interaction  in  this  program 

consists  of  pressing  the  space  bar  approximately  25 
times  and  entering  correct  multiple  choice 
responses  in  the  drill  section  (5  questions). 

Example:  Student  interaction  in  this  program 

consists  of  menu  selections,  page  turning  devices 
and  answer  inputs  in  the  drill  section.  In  the  menu, 
the  student  is  asked  to  enter  the  name  of  the 
program  section  desired.  During  the  explanation, 
(10  pages),  the  student  can  go  forward  or  backward 
by  using  the  right  or  left  arrow  keys.  The  drill 
questions  require  the  student  to  enter  the  correct 
numerical  answer  along  with  the  % sign. 

Describe  the  manner  in  which  the  program  selects 
questions.  (For  example,  randomly,  randomly  from  a 
limited  data  bank,  in  a preprogrammed  order,  in  a 
predictable  sequence,  from  a teacher  created  file,  etc.) 

Describe  the  feedback  techniques.  Some  questions  that 
may  be  answered  are: 

1.  When  is  feedback  provided?  (after  all  user 
responses,  when  user  response  is  wrong,  randomly 
on  correct  responses,  etc.) 

2.  What  type  of  feedback  is  provided?  (e.g.  graphics, 
text,  audio,  combination,  etc.) 

3.  What  content  is  included  in  the  feedback?  (e.g. 
general  message,  specific  hints  depending  upon  the 
error,  the  correct  answer,  a restatement  of  the 
correct  answer,  explanation  of  incorrect  response, 
presentation  of  new  information,  etc.) 

4.  What  is  the  extent  of  the  variety  of  the  feedback? 
(e.g.  The  same  phrase  "try  again"  is  used  after 
each  incorrect  response.) 

5.  Is  quantitative  feedback  provided?  (e.g.  You  got  9 
questions  correct  out  of  20.) 

Example:  There  are  two  types  of  feedback 

employed.  On  inputting  a correct  answer,  the 
student  receives  a brief  congratulatory  message 
chosen  randomly  from  approximately  10 
statements.  If  an  incorrect  response  is  entered, 
the  student  is  first  told  to  "try  again".  If  (s)he  is 
wrong  on  the  second  attempt,  the  correct  answer  is 
shown.  Quantitative  feedback  is  given  at  the  end 
of  the  drill. 
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Evaluation 

Techniques 


Branches 


Types  of 
Control 


Example:  Graphical  and  audio  feedback  are 

employed  throughout  the  program.  Correct 
answers  are  rewarded  with  an  extensive  graphical 
picture  (4  pictures  available).  Incorrect  answers 
are  punished  by  a buzzing  sound  and  a large  "X" 
which  flashes  on  and  off  the  screen. 

Describe  the  evaluation  techniques  that  are  used  to 
determine  the  student’s  success  in  the  program. 

1.  Is  there  a pre-test  or  a post-test? 

2.  What  are  the  criteria  for  success?  Who  sets  the 
criteria? 

Note  that  within  this  context,  a drill  is  not  an 
evaluation  technique.  A drill  presents  questions,  often 
in  a structured  way,  to  give  the  user  practice  in  a 
particular  skill.  Often,  more  than  one  attempt  at  a 
question  is  given  and  hints  may  also  be  provided. 
However,  a test  determines  if  the  practice  of  the  skill, 
the  involvement  of  the  student  in  the  program,  has  been 
or  is  likely  to  be  successful.  Questions  are  not 
structured  to  lead  a student  towards  more  difficult 
questions,  only  one  chance  is  given  and  feedback  is 
limited  to  an  indication  of  right  or  wrong. 

Describe  the  branches  that  exist  within  the  program. 
What  types  of  branches  are  available  to  the  user?  Is 
the  program  linearly  sequenced  or  is  there  provision  for 
review,  remediation,  enrichment,  help,  etc.? 

Describe  the  type  of  control  that  is  used  with  each  of 
the  following:  Sequence,  Rate  and  Amount. 

There  are  three  types  of  control  - PROGRAM, 
TEACHER  and  USER.  Program  control  is  based  on  pre- 
set criteria  built  into  the  program  while  teacher  and 
user  controls  are  based  on  entries  made  during  the 
operation  of  the  program.  If  teacher  or  student  control 
is  possible,  describe  how  this  is  done  (i.e.  through  a 
menu,  management  system,  changing  the  program 
listing,  etc.). 

Sequence  refers  to  the  path  of  the  user  through  the 
content  described  earlier.  For  example:  Sending  a 

student  to  a remedial  loop  automatically  if  (s)he  makes 
a certain  number  of  errors  is  program  control.  User 
control  gives  the  student  the  opportunity  to  go  to  a 
particular  part  of  the  program,  e.g.  skip  the  explanation 
and  go  straight  to  the  drill.  Teacher  control  allows  the 
teacher  to  specify,  before  a student  begins  the 
program,  what  parts  of  the  program  (s)he  is  to  do. 
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INSTRUCTIONAL 

FORMAT 

EVALUATION 


Student 

Interaction 


Rate  refers  to  the  time  allowed  to  read  each  display 
and  to  answer  questions. 

Amount  refers  to  the  amount  of  material  that  is 
presented  in  any  one  branch  and  how  it  is  controlled. 
For  example,  the  number  of  questions  done,  the  number 
of  examples  seen,  the  number  of  pages  read  or  re-read. 
Indicate  the  presence  of  any  other  user/teacher  control 
features.  (For  example:  timing  of  input,  type  of 

feedback,  frequency  of  feedback,  randomization  of 
feedback,  content  of  feedback,  mastery  level,  speed  of 
character  generation,  content  of  the  post  test.) 

Example: 

The  sequence  is  controlled  by  the  user  through  menu 
selections.  Once  into  the  program  however,  the 
sequence  is  controlled  by  the  program. 

The  rate  is  controlled  by  the  user  by  pressing  the  space 
bar. 

The  amount  is  controlled  by  the  program  or  by  the 
teacher.  Students  will  generally  all  do  twenty 
questions,  however,  the  management  system  allows  the 
teacher  to  set  the  number  from  10-30  for  individual 
students  or  for  the  class  as  a whole. 


Write  narrative  statements  which  will  evaluate  the 
instructional  format  described  earlier.  The  following 
questions  are  provided  to  assist  you  in  supplying 
evaluative  comments. 

1.  Is  the  student  interaction  appropriate  for  the 

target  audience? 

- Can  the  target  audience  interact  with  the 
program  easily? 

- Does  the  method  of  interaction  detract  the 
user's  attention  from  the  learning  process? 

- Does  the  time  required  to  complete  the  program 
exceed  the  attention  span  and/or  physical 
endurance  of  the  intended  users? 

- If  there  is  a time  limit,  is  the  concept  of  timing 
appropriate  to  the  target  audience? 

2.  Does  the  program  contain  a method  of  inquiry  that 

promotes  learning? 

- Is  the  student  encouraged/permitted  to  learn 
through  manipulation  of  the  content  rather  than 
by  simply  passively  reviewing  facts?  (e.g. 
manipulating  a variable  to  verify  or  formulate  a 
theory.) 
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Questioning 

Techniques 


F eedback 
Techniques 


3.  Is  the  student  interaction  effective? 

- Is  there  sufficient  interaction  to  meet  the 
intents  of  the  program?  Does  the  student 
participate  or  just  watch? 

- Does  the  interaction  promote  learning? 

- Is  the  type  of  interaction  appropriate  for  the 
content  and  the  intents  of  the  program?  (e.g. 
using  multiple  choice  inputs,  true/false,  etc.) 

- Is  the  type  of  interaction  consistent? 

- Is  the  student  given  appropriate  directions  as  to 
the  specific  forms  of  input  needed? 

- Is  the  program  tolerant  in  accepting  student 
inputs?  (e.g.  "YM  instead  of  "YES",  slight 
misspellings,  equivalent  numerical  answers  such 
as  3.0  for  3,  etc.) 

- Is  the  time  limit  appropriate?  Can  it  be 
controlled  by  the  student/teacher? 

- Is  the  entry  of  the  user's  input  apparent  to  the 
user?  Is  it  acknowledged  where  necessary? 

- Does  the  program  respond  satisfactorily  to 
unexpected  entries  of  space  bar,  alphabetic  or 
numeric  inputs? 

- If  the  program  is  designed  for  paddle  inputs,  does 
it  also  allow  keyboard  entry?  Is  a check  made 
for  type  of  entry  to  be  used? 

- Where  appropriate,  does  the  program  contain  a 
calculator  function  and/or  can  the  user  work 
questions  out  on  the  screen? 

- Does  the  program  allow  correction  of  incorrect 
entries?  Can  this  be  easily  done? 

- Does  the  program  accept  input  at  an  appropriate 
rate? 

4.  Are  questions/questioning  techniques  appropriate 

to  the  target  audience? 

3.  Is  the  questioning  technique  effective? 

- Are  questions  appropriate  to  the  content? 

- Are  questions  effectively  randomized,  placed  in 
a fixed  sequence,  drawn  appropriately  out  of  a 
data  bank  or  repeated  where  necessary? 

- Are  cueing  techniques  (hints/prompts/faded 
prompts,  etc.)  used  effectively? 

- Are  the  number  of  trials  appropriate? 

6.  Is  the  feedback  appropriate  to  the  target  audience? 

- Is  the  form  of  the  feedback  (graphics,  reading 
level,  sound,  etc.)  appropriate  to  the  target 
audience? 

- Is  the  content  of  the  feedback  appropriate? 


DISK  EVALUATION  FORM 


ALBERTA  EDUCATION  Version  2.0 


DESCRIPTION 

Disk  Title:  

Publisher: 
Subject: 
Topics: 


Target  Audience:  

Components:  Cost: 


System  Requirements:  (Beyond  48K,  Applesoft,  1 disk  Drive,  DOS  3.3.,  B&W  Monitor) 


IMPLEMENTATIONAL  SUPPORT  DESCRIPTION 

Describe  the  entry  competencies  (if  any)  (Guidebook,  page  18) 


Describe  the  management  system  (if  any)  (Guidebook,  page  18) 


Describe  the  teacher  documentation  (Guidebook,  page  19) 


Describe  the  user  support  materials  (if  any)  (Guidebook,  page  19) 


I MPLEMENT ATION  AL  SUPPORT  EVALUATION  (Guidebook,  page  19) 


OBJECTIVES  DESCRIPTION  (Guidebook,  page  20) 


PROGRAM  EVALUATION  FORM 


ALBERTA  EDUCATION  Version  2.0 

PROGRAM  NAME(S): TARGET  AUDIENCE(S): 


PEDAGOGICAL  CONTENT  DESCRIPTION  (Guidebook,  page  4) 


PEDAGOGICAL  CONTENT  EVALUATION  (Guidebook,  page  5) 


INSTRUCTIONAL  FORMAT  DESCRIPTION 

Describe  the  student  interaction  in  the  program:  (Guidebook,  page  S) 


Describe  the  questioning  techniques  in  the  program:  (Guidebook,  page  9) 


Describe  the  feedback  techniques  used  in  the  program:  (Guidebook,  page  9) 


Describe  the  evaluation  techniques  used  in  the  program:  (Guidebook,  page  10) 


Describe  the  branches  available  in  the  program:  (Guidebook,  page  10) 


Describe  the  types  of  control  in  the  program: 


(Sequence,  Rate,  Amount)  (Guidebook,  page  10) 


INSTRUCTIONAL  FORMAT  EVALUATION  (Guidebook,  page  11) 


TECHNICAL  CHARACTERISTICS  DESCRIPTION 

(Guidebook,  page  15) 


TECHNICAL  CHARACTERISTICS  EVALUATION  (Guidebook,  page  15) 


OBSERVATIONS  OF  STUDENT  USERS  (Guidebook,  page  16) 


Name  of  Evaluator: 


EVALUATION  OF  THE  OBJECTIVES  OF  THE  DISK  (Guidebook,  page  21) 


SUMMARY  EVALUATION  OF  THE  PEDAGOGICAL  CONTENT  OF  THE  DISK 

(Guidebook,  pages  5 and  23) 


SUMMARY  EVALUATION  OF  THE  INSTRUCTIONAL  FORMAT  OF  THE  DISK 

(Guidebook,  pages  11  and  2k) 


SUMMARY  EVALUATION  OF  THE  TECHNICAL  CHARACTERISTICS  OF  THE  DISK 
(Guidebook,  pages  15  and  2k) 


SUMMATION:  Major  Strengths  and  Weaknesses  (Guidebook,  page  22) 


SUGGESTIONS  TO  THE  PRODUCER  (Guidebook,  page  22) 


RECOMMENDATION 


This  disk  should  be  evaluated  further  to  determine  if 

"PRESCRIBED"  Learning  Resource  is  warranted. 

This  disk  should  be  evaluated  further  to  determine  if 

"RECOMMENDED"  Learning  Resource  is  warranted. 

This  disk  should  be  evaluated  further  to  determine  if 

"SUPPLEMENTARY"  Learning  Resource  is  warranted. 

This  disk  should  be  returned  to  the  publisher. 

I agree  that  the  copyright  to  this  report  rests  with  Alberta  Education  and  confirm  that  no 
attempt  was  made  to  copy  or  modify  the  program  or  the  accompanying  documentation. 

Name  of  evaluator: 


Date: 

Signature: 


authorization  as  a 
authorization  as  a 
authorization  as  a 


This  evaluation  was  done  for  internal  purposes  only.  It  is  not  to  be  reproduced  in 
part  or  in  whole  without  the  express  written  permission  of  Alberta  Education. 
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Evaluation 


7.  Is  the  feedback  effective? 

- Are  user  inputs  accurately  evaluated  as  right  or 
wrong? 

- Does  learning  take  place  regardless  of  the 
student's  response?  For  example,  positive 
feedback  can  reinforce  while  negative  feedback 
corrects. 

- Does  the  feedback  distinguish  between  a wrong 
answer  or  the  wrong  format? 

- Is  the  feedback  non-threatening,  immediate  and 
positive?  Is  the  program  "user-sensitive"? 

- Is  the  feedback  corrective  where  necessary? 

- Is  the  feedback  relevant  to  a student's  history  of 
responses? 

- Are  traps  set  to  catch  both  format  and 
knowledge  errors? 

- Does  the  feedback  explain  why  a response  is 
wrong? 

- Are  cues  or  prompts  used  after  a wrong 
response? 

- Is  the  feedback  to  a wrong  response 

unnecessarily  attractive? 

- Can  the  user/teacher  control  the  feedback  (e.g. 
type,  amount,  when  given)? 

- Can  the  user  compare  his/her  response  to  the 
correct  answer? 

- Does  the  feedback  remain  on  the  screen  for  an 
appropriate  amount  of  time? 

- Is  the  feedback  personalized? 

- Is  the  feedback  randomized  where  appropriate? 

- Is  the  feedback  placed  on  the  screen 

appropriately?  consistently? 

- Is  the  feedback  boring,  repetitive,  time 

consuming  or  unnecessarily  detailed? 

- Is  negative  feedback  made  inappropriately  public 
through  the  use  of  sound? 

- Is  quantitative  feedback  available  where 
necessary? 

- Does  quantitative  feedback  provide  the 

user/teacher  with  useful,  understandable 

information? 

8.  Are  pre-tests/post-tests  appropriately  present/ 

absent? 

9.  If  present,  are  tests  appropriate  to  the  target 

audience? 

- Are  the  criteria  for  success  appropriate  for  the 
ability/skills  of  the  intended  user? 

- Can  the  teacher  set  such  criteria? 

10.  If  present,  are  the  evaluation  techniques  effective? 

- Does  the  test  measure  the  user's  achievements 
on  all  the  intents? 

- Does  the  test  content  accurately  reflect  the 
content  of  the  material? 
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- Do  test  results  provide  sufficient  information  to 
the  user/teacher  (e.g.  breakdown  of  scores  by 
objectives)  to  enable  efficient  followup? 

Are  branches  appropriately  present/absent? 

If  present,  are  the  branches  appropriate  for  the 
target  audience? 

If  present,  are  the  branches  effective  in 
accommodating  individual  differences?  Should 
there  be  more?  Should  there  be  less? 

- Are  entry/exit  points  effectively  controlled? 
(e.g.:  based  on  user  success  or  lack  of  success, 
user  volition,  teacher  discretion,  etc.) 

Does  the  teacher/user  have  an  appropriate  amount 
of  control  over  the  program? 

- Where  desirable,  does  the  teacher  have  control 
over  features  other  than  sequence,  rate  and 
amount?  (E.g.  imposition  or  alterability  of 
timed  input,  type  of  feedback,  randomization  of 
feedback,  feedback  content,  mastery  level, 
speed  of  character  generation,  content  of  the 
pre-test  and  post-test,  addition  and/or 
adaptation  of  content  units,  recording  of  scores.) 

Are  control  features  which  are  present  effectively 
designed? 

- Does  the  user  have  control  of  the  program  where 
appropriate?  E.g.  Can  the  student  choose  the 
path(s)  (s)he  wishes  to  follow? 

- If  teacher/user  control  is  available,  are  such 
features  easily  implemented?  Are  default 
options  available? 

C.  TECHNICAL  CHARACTERISTICS 

Technical  characteristics  sections  focus  on  the  program’s  output  as  well  as 
on  its  'user  friendliness'.  To  be  described  are  such  features  as  displays, 
special  features  (e.g.  color,  sound  and  graphics)  and  ease  of  use. 

Effective  screen  DISPLAYS  are  needed  to  allow  the  user  to  learn  the 
pedagogical  content  described  earlier.  Examples  of  poor  displays  are 
incorrect  grammar,  spelling  and  punctuation,  inappropriate  character  size, 
excessive  amounts  of  information  crammed  into  each  screen  display  and 
insufficient  spacing  between  lines  of  text. 

Effective  use  of  COLOR,  GRAPHICS  and  SOUND  can  be  important  in 
motivating  the  user  and  adding  to  his/her  understanding  of  the  content. 
However,  these  features  should  be  integral  to  the  design  of  the  program 
such  that  their  absence  would  seriously  limit  the  program's  usefulness  in  the 
classroom.  Unfortunately,  these  features  are  often  used  solely  to  add  'cute' 
positive  feedback  and  their  inclusion  is  less  than  helpful. 


Branches  11. 

12. 

13. 

Types  of  14. 

Control 


15. 
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Common  problems  include  generation  of  graphical  rewards  that  are  time 
consuming  and  which  become  boring  and  unmotivational  with  repeated  use; 
sound  that  distracts  and/or  draws  attention  to  a user's  errors;  ineffective 
depiction  of  an  object/idea  (e.g.  a heart  made  up  of  low  resolution 
rectangles). 

STUDENT  EASE  OF  USE  is  important  if  the  program  is  to  be  effective  in 
the  classroom.  Programs  which  are  difficult  for  the  user  to  operate  create 
problems  in  the  classroom  as  the  teacher  is  constantly  being  asked  for  help. 
Alternatively,  the  student  can  become  frustrated  with  trying  to  find  his/her 
way  through  the  program  and  will  fail  to  achieve  the  objectives.  Examples 
of  poor  programs  include  those  that  provide  no  screen  instructions  on  use, 
provide  no  indication  to  the  user  as  to  where  (s)he  is  in  the  program,  do  not 
allow  easy  exit  and  require  more  attention  from  the  user  in  operating  the 
program  than  in  actually  learning  the  content. 

TECHNICAL  Write  narrative  statements  that  will  describe  the 

CHARACTERISTICS  type  and  size  of  print  used.  Explain  the  extent  to 
DESCRIPTION  which  color,  sound  and  graphics  are  used.  (Indicate  if 

the  evaluation  was  done  without  a color  monitor.) 

TECHNICAL  Write  narrative  statements  which  will  evaluate 

CHARACTERISTICS  the  technical  characteristics  described  earlier. 
EVALUATION  The  following  questions  are  provided  to  assist  you  in 

supplying  evaluative  comments. 

Displays  1.  Are  displays  appropriate  for  the  target  audience? 

- Is  character  size  appropriate? 

- Is  the  font  appropriate? 

- Are  upper  and  lower  case  letters  used  where 
appropriate? 

2.  Are  displays  effective? 

- Are  they  free  from  grammar,  spelling, 
punctuation  and  hyphenation  errors? 

- Are  characters  well  formed? 

- Is  the  amount  of  material  presented  at  one  time 
appropriate?  Is  it  too  lengthy  or  wordy? 

- Is  the  material  clear  and  easy  to  read?  Is  there 
adequate  spacing  for  clarity? 

- Are  message  design  principles  used  to  place 

emphasis  on  the  important  concepts?  Are 

paging  and/or  selective  erasing  used 

appropriately? 

- Is  the  transition  between  displays  effective? 

- Are  special  features  such  as  flash,  inverse, 
scrolling,  split  screen,  etc.  used  appropriately 
and  effectively? 

- Is  there  a clear  indication  when  and  where  input 
is  required? 
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- Are  text  screens  presented  at  an  appropriate 
rate? 

- Are  text  screens  erased  at  a rate  so  that  the 
user  is  given  an  appropriate  amount  of  time  to 
read  and  absorb  the  information? 

- Does  the  program  accept  both  upper  and  lower 
case  inputs  or  does  the  program  provide  capitals 
where  necessary? 

- Can  the  user  review  previous  screen  displays? 

- Is  text  protected  by  "word  wrap"? 

3.  Are  color,  graphics  and  sound  appropriate  for  the 
target  audience? 

k.  Are  color,  graphics  and  sound  used  effectively? 
Does  their  use  add  to  or  detract  from  the 
effectiveness  of  the  instruction? 

- Are  they  used  appropriately  for  the  content? 

- Are  they  appropriately  mixed  with  text  so  as  to 
give  variety  to  the  presentation? 

- Is  the  use  of  these  features  motivational? 

- Will  the  sound  distract  others  in  the  classroom? 
Can  it  be  suitably  controlled? 

- If  color  is  present,  will  the  program  be  effective 
on  a monochrome  monitor? 

- Do  the  graphics  effectively  portray  the  intended 
object/idea? 

- If  animation  is  present,  does  it  add  to  or  detract 
from  the  effectiveness  of  the  instruction? 

Ease  of  Use  5.  Can  the  intended  user  easily  and  independently 
operate  the  program? 

- Does  the  operation  of  the  program  interfere  with 
the  user’s  concentration? 

- Is  the  user  aware  of  where  (s)he  is  in  the 
program?  (e.g.  question  number,  page  headings) 

- Can  the  user  exit  from  the  program,  return  to 
the  menu  or  move  to  another  section  easily  when 
necessary? 

- Can  the  user  exit  from  the  program  and,  if 
appropriate,  return  to  that  point  later? 

- Are  instructions/help  avenues  available  at 
appropriate  points  in  the  program  and  are  they 
appropriate  for  the  target  audience? 

- Is  the  program  reliable  and  bomb-proof  under 
normal  use? 


Color 

Graphics 

Sound 


D.  OBSERVATIONS  OF  STUDENT  USERS 

Summarize  the  results  of  student  use  of  the  program.  This  could 
indicate  the  evaluator's  observation  of  the  user(s)  as  well  as  comments 
and  observations  made  by  the  user(s).  Ensure  that  the  test  group  is 
adequately  described. 
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ra.  DISK  EVALUATION  FORM 


A.  DESCRIPTION 

This  section  contains  brief  descriptive  data  on  the  product  for  the 
information  of  evaluators.  This  information  and  additional  descriptions  on 
the  formal  reports  are  entered  by  Clearinghouse  staff. 


Title 

The  title  of  the  program,  disk  or  series. 

Publisher 

Name  of  the  company  publishing  the  product. 

Subject 

Subject  area,  e.g.  math,  science,  language  arts. 

Target 

Audience 

Age,  grade  level  or  other  ability  level  descriptors  of  the 
target  audience  for  which  the  package  has  been 
designed  or  would  be  appropriate. 

Topics 

Subject  area  topics  included  in  the  package. 

Components 
and  Costs 

The  physical  components  of  the  package,  i.e.  disk(s), 
guide(s),  etc.  plus  the  package/component  costs. 

System 

Requirements 

The  hardware  or  software  that  is  required  beyond  the 
standard  package  of  ^8K,  1 disk  drive,  3.3  DOS  and  a 
monochrome  monitor. 

B.  IMPLEMENTATIONAL  SUPPORT 

The  IMPLEMENTATIONAL  SUPPORT  section  focuses  on  the  materials 
and  management  functions  that  are  available  to  the  teacher  who  wishes 
to  implement  the  program.  Descriptions  in  this  section  summarize  any 
prerequisite  skills,  the  scope  of  any  management  functions  and  the 
content  of  any  teacher  documentation  and/or  user  support  materials. 
The  evaluative  comments  focus  on  the  ease  with  which  the  package  can 
be  implemented  in  the  classroom. 

An  effective  MANAGEMENT  SYSTEM  will  efficiently  and  accurately 
track  student  progress  and  will  provide  the  teacher  with  important 
information  that  will  enable  him/her  to  modify  the  student's  program. 
Common  problems  are: 

- providing  scorekeeping  with  only  the  most  recent  score  recorded; 

- having  limited  capacity:  e.g.  a capacity  of  '50'  may  in  fact  be  50 

scores,  not  50  students; 

- storing  cumulative  percentages  only  and  not  giving  any  information 
on  the  number  of  questions  attempted  or  the  history  of  attempts; 

- not  providing  any  diagnosis  or  prescription; 

- making  the  system  difficult  to  understand  and  master; 

- making  the  system  difficult  to  use  in  the  classroom; 

- not  making  records  alphabetical  and  modifiable. 
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Sufficient  detail  in  the  TEACHER  DOCUMENTATION  can  ease  the 
difficulty  of  implementation  by  providing  the  teacher  with  necessary 
information  on  the  package  as  well  as  by  providing  advice  on  its  use. 
An  important  ingredient  in  the  assessment  is  a statement  on  whether 
the  support  materials  are  sufficient  as  well  as  an  evaluation  of  the 
quality  of  those  that  are  present.  Another  aspect  to  the  ease  of  using  a 
package  in  a classroom  is  the  amount  of  learning/interaction  that  is 
going  to  be  required  from  the  teacher.  Some  educational  software 
packages,  just  because  of  their  design  complexity,  require  extensive 
preparation  by  the  teacher  and  frequent  supervision  during  use.  An 
evaluative  statement  assessing  the  difficulties  that  the  teacher  is  going 
to  have  while  using  the  software  is  useful  information  for  prospective 
buyers. 

Providing  USER  SUPPORT  MATERIALS  such  as  drill  sheets,  workbook 
activities,  additional  reading,  etc.  can  also  ease  the  implementation  of 
a package  in  the  classroom.  Evaluative  comments  focus  again  on 
whether  sufficient  materials  have  been  provided  as  well  as  on  whether 
those  which  are  available  are  integral  to  the  design  of  the  package. 

IMPLEMENTATIONS 

SUPPORT  Write  narrative  statements  which  will  describe  the 
DESCRIPTION  following  aspects  of  implementational  support. 

Entry  Describe  any  special  prerequisites  that  are  beyond  the 

Competencies  normal  grade  level  skills. 

Describe  also  any  student  activities  that  should  be 
completed  prior  to  beginning  the  work  on  the  disk. 

Indicate  if  the  programs  on  this  disk  are  prerequisites 
for  other  programs  on  the  disk  or  for  programs  on  other 
disks  in  the  series. 

Management  Describe  the  functions  available  in  the  management 
System  system.  These  may  include: 

1.  Class  lists: 

e.g.  initiating,  maintaining,  adding  to,  deleting 
from; 

2.  Reports: 

e.g.  by  individual  student,  by  class,  by  specific 
unit; 

3.  Prescription: 

e.g.  prescription  of  specific  assignments  or  a range 
of  assignments  for  individuals  or  for  the  class; 

4.  Diagnosis: 

diagnosis  of  a student's  weaknesses; 

3.  Security: 

passwords,  etc. 
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Teacher 

Documentation 

6.  Option  of  use: 

Does  the  teacher  have  the  option  of  using  the 
program  as  an  unmanaged  activity  for  any 
student? 

Also,  describe  the  capacity.  How  many  students  can 
be  kept  in  one  class  list?  How  many  scores  can  be 
kept  for  one  student?  How  many  classes  can  be  kept? 

Describe  the  contents  of  the  documentation  (both 
on  the  disk  and  on  associated  print  materials).  The 
documentation  may  include  any  of  the  following: 
instructional  objectives; 
suggested  grade,  age  or  ability  levels; 
required  reading  level  and  other  prerequisite  skills; 
field  test  results; 
rationale; 

strategies  for  integration; 

suggested  pre-activities  and/or  post-activities; 
explanation  of  management  system/special  control 
features; 

sample  program  run; 
content  scope  and  sequence; 
program  operating  instructions. 

User  Support 
Materials 

Describe  the  extent  of  any  STUDENT  materials 
such  as  pre-instruction  activities,  work  sheets,  follow- 
up activities,  etc.  Indicate  if  the  materials  are 
integral  to  the  design  of  the  package  or  if  they  are 
optional. 

IMPLEMENTATION  AL 

SUPPORT  Write  narrative  statements  which  will  evaluate 


EVALUATION 

the  implementational  support  described  earlier.  The 
following  questions  are  provided  to  assist  you  in 
supplying  evaluative  comments: 

Entry 

Compentencies 

1.  Are  the  entry  competencies  which  have  been 
provided  in  the  package  appropriate  for  the 
target  audience? 

2.  Have  all  the  entry  competencies  which  are 
required  by  the  target  audience  been  identified? 

Management 

System 

3.  Do  the  functions  work  as  described? 

Is  the  system  effective  in  meeting  the  needs  of 
the  teacher? 

5.  How  easy  is  the  system  to  set  up  and  use? 

6.  Is  the  capacity  sufficient? 

7.  Are  data  easily  and  effectively  collected? 
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Teacher 

8. 

Does  the  documentation  contain  all  the  features 

Documentation 

needed  to  make  this  program  easy  to  use  in  the 
classroom? 

9. 

Is  the  documentation  well  laid  out  and  easy  to  use? 

User 

10. 

Are  user  support  materials  appropriate  for  the 

Support 

target  audience? 

Materials 

11. 

Are  user  support  materials  effective? 

- Do  they  contribute  to  the  learning  process? 

- Are  these  materials  sufficient? 

Ease  of 

You 

should  also  consider  the  manner  in  which  the  disk 

Implementation  can  be  used  in  the  classroom.  Some  questions  to  be 

addressed  are: 

12.  Is  the  disk  adaptable  to  a variety  of  classroom 
learning  environments? 

13.  Does  the  disk  require  a minimal  amount  of 
equipment  manipulation? 

14.  Can  the  disk  be  used  by  a teacher  with  a minimum 
of  computer  competency? 

15.  Do  the  students  require  a minimum  amount  of 
teacher  supervision  when  using  the  program? 

16.  Are  error  handling  and  identification  procedures 
sufficiently  detailed  so  that  a teacher  can  help  a 
student? 

17.  Are  excessive  additional  demands  placed  on  the 
teacher’s  time  to  learn  to  use  the  package  and  to 
operate  it  within  a classroom  setting? 

C.  OBJECTIVES 

Much  of  the  evaluation  of  a courseware  package  is  a determination  of 
whether  the  program  meets  its  intents.  As  such,  stated  OBJECTIVES  are 
a very  necessary  part  of  the  courseware  documentation.  Unfortunately,  a 
common  fault  with  much  of  the  educational  software  currently  on  the 
market  is  either  the  complete  absence  of  stated  progam  objectives  or  the 
lack  of  specificity  in  their  statement.  Evaluation  statements  should  first 
address  the  manner  in  which  the  objectives  are  presented  and  then 
comment  on  whether  the  objectives  were  reasonable  in  view  of  the  target 
audience,  the  medium  chosen  and  the  design  of  the  program. 
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Description  List  the  objectives  that  the  program  is  intended  to 
achieve  and  indicate  if  they  are  stated  or  inferred. 

Objectives  are  the  behaviours  that  the  learner 
manifests  as  a result  of  involvement  with  the  materials. 
The  evaluator  should  ask  himself  what  it  is  the  learner 
will  be  able  to  do  after  using  these  materials. 

Example:  Given  line  drawings  of  geometric  figures 

(circle,  ellipse,  square,  rectangle,  parallelogram, 
triangle  and  diamond),  presented  in  a straight  line,  four 
at  a time,  (three  identical,  one  different;  randomly 
located),  the  learner  will  indicate  which  one  is 
different.  Given  three  trials  where  each  figure  is 
compared  with  each  of  the  others,  the  learner  should 
obtain  at  least  two  out  of  three  correct  for  each  figure 
and  no  less  than  50  out  of  60  correct  for  all  figures. 

Example:  Given  a number  of  random  representations  of 
various  objects,  (e.g.  a hand,  shoe,  pencil  or  book) 
shown  over  or  under  a referent  object,  the  learner  will 
indicate  if  the  object  is  "over"  or  "not  over"  the 
referent  object. 

Evaluation  Write  narrative  statements  which  will  evaluate  the 
objectives  described  earlier.  The  following  questions 
are  provided  to  assist  you  in  supplying  evaluative 
comments. 

1.  Are  the  developer's  objectives  clearly  stated? 

2.  Are  they  appropriate  to  the  target  audience? 

3.  Are  they  appropriate  to  the  medium? 

4.  Will  satisfactory  completion  of  the  program  result 
in  their  fulfillment? 


D.  DISK  EVALUATION  SUMMARIES 

Evaluative  statements  in  this  section  cover  the  entire  disk.  If  two  or  more 
Program  Evaluation  Forms  have  been  completed,  this  section  will  include 
summaries  of  the  evaluative  comments  made  on  those  forms.  If  only  one 
Program  Evaluation  Form  has  been  completed,  the  reader  should  be  referred 
to  evaluation  statements  made  on  that  form. 

Disk  Pedagogical  Content  - see  page  5 of  this  guidebook. 

Disk  Instructional  Format  - see  page  11  of  this  guidebook. 

Disk  Technical  Characteristics  - see  page  15  of  this  guidebook. 
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E.  SUMMATION 

Summarize  the  strengths  and  weaknesses  of  the  disk.  Considering  the 
comments  made  in  the  sections  above,  what  is  your  evaluation  of  the  entire 

disk? 


F.  SUGGESTIONS  TO  THE  PRODUCER 

Indicate,  in  priority  order,  the  changes  that  the  producer  could  make  in 
order  to  improve  the  quality  of  the  diskette.  The  suggestions  should  be 
specific.  Comments  such  as  "The  package  should  be  redesigned"  or  "Improve 
the  motivational  aspects  of  the  material"  are  of  little  use  to  the  producer 
and  should  be  avoided. 


G.  RECOMMENDATION 

Check  one  recommendation.  See  the  guidelines  for  making 
recommendations  on  page  23  of  this  guidebook. 
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IV.  GUIDELINES  FOR  MAKING  RECOMMENDATIONS 

There  are  no  hard  and  fast  rules  with  which  to  equate  the  strengths  and 
weaknesses  of  a disk  to  "the  correct  recommendation".  Recommendations 
are  subjective  and  will  vary  from  one  evaluator  to  another.  We  hope  that 
the  degree  of  variance  will  be  small  and  that  the  process  of  synthesizing 
three  evaluations  will  minimize  any  differences  that  might  occur. 

However,  to  keep  these  variances  small  and  to  strive  for  consistency  across 
subject  and  grade  levels,  it  is  advantageous  to  consider  some  guidelines. 

Within  this  instrument,  there  are  two  questions  which  have  been  addressed 
in  almost  every  evaluation  section: 

1.  the  appropriateness  to  the  target  audience,  and 

2.  the  effectiveness  in  meeting  the  intents  of  the  program. 

A program  that  is  ineffective  should  be  rejected.  On  the  other  hand,  a 
program  which  is  inappropriate  to  its  target  audience  may  still  be  effective 
with  another  target  audience  and  could  be  considered  for  limited  use 
(supplementary)  in  the  classroom. 

What  makes  a program  ineffective  or  inappropriate?  Certainly  the  degree 
of  the  weakness  has  to  be  considered.  A program  should  not  be  rejected 
because  of  a minor  flaw.  The  number  of  weaknesses  should  also  be 
considered  and  a program  with  4 or  5 minor  flaws  may  have  its  use 
restricted  because  of  the  cumulative  effects  of  small  weaknesses. 

Major  flaws  are  those  which  seriously  hinder  the  learning  that  is  supposed  to 
take  place  in  a program.  They  are  generally  in  the  areas  of  "pedagogical 
content"  and  "instructional  format".  Minor  flaws  are  those  which  may 
slightly  detract  from  the  program's  effectiveness.  They  are  generally  in  the 
area  of  "technical  characteristics"  or  are  weaknesses  in  meeting  the  needs 
of  the  specified  users.  One  minor  flaw  by  itself  however  is  not  necessarily 
cause  for  a lower  rating. 

The  following  may  be  considered  as  'major  flaws'  in  a tutorial  or  in  a drill 
and  practice  program.  Programs  which  have  any  one  of  the  features 
described  below  should  be  carefully  considered  if  advancement  is 
contemplated.  Only  in  exceptional  cases  should  programs  which  have  any 
two  of  these  weaknesses  be  advanced. 

PEDAGOGICAL  CONTENT 

1.  The  pedagogical  content  is  outside  of  the  Alberta  curriculum. 

2.  The  program  attempts  to  cover  too  much  in  one  sitting. 

3.  Incremental  steps  within  the  sequence  are  too  large  for  the  intents  of 
the  program  and  the  target  audience. 

4.  The  sequence  from  easy  to  hard  and/or  from  concrete  to  abstract  has 
not  been  sufficiently  developed  (for  developmental  as  opposed  to  review 
programs). 
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5.  There  is  no  instructional  component  or  there  is  insufficient  detail  in  the 
instruction  (for  programs  where  this  need  is  appropriate). 

6.  There  is  insufficient  practice. 

7.  The  program  lacks  clarity  (terminology,  reading  levels,  levels  of 
abstractions,  etc.). 

8.  The  content  is  inaccurate. 

INSTRUCTIONAL  FORMAT 

9.  Students  read  too  much  and/or  interact  too  little  with  the  program. 

10.  Question  randomization  (where  necessary  for  the  intents  of  the 
program)  is  ineffective. 

11.  User  inputs  are  not  accurately  evaluated  as  right  or  wrong. 

12.  There  is  insufficient  assistance  given  as  part  of  the  negative  feedback. 

13.  The  user/teacher  does  not  have  appropriate  control  over  the  program.. 

TECHNICAL  CHARACTERISTICS 

14.  The  disk  has  programming  errors. 

IMPLEMENTATIONAL  SUPPORT 

13.  The  program  is  excessively  difficult  for  the  teacher  to  Implement 
within  a classroom  situation. 
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V.  DEFINITIONS 


PRESCRIBED  LEARNING  RESOURCES  are  those  "learning  resources 
authorized  by  the  Minister  as  being  most  appropriate  for  meeting  the 
majority  of  goals  and  objectives  for  courses,  or  substantial  components  of 
courses,  outlined  in  provincial  Programs  of  Study". 


RECOMMENDED  LEARNING  RESOURCES  are  "those  learning  resources 
authorized  by  Alberta  Education  because  they  complement  Prescribed 
Learning  Resources  by  making  an  important  contribution  to  the  attainment 
of  one  or  more  of  the  major  goals  of  courses  outlined  in  provincial  Programs 
of  Study". 


SUPPLEMENTARY  LEARNING  RESOURCES  are  "those  additional  learning 
resources  authorized  by  Alberta  Education  to  support  courses  outlined  in  the 
provincial  Programs  of  Study  by  reinforcing  or  enriching  the  learning 
experience". 

This  category  should  not  be  a dumping  ground  for  all  courseware  that  is  less 
than  "Recommended".  Courseware  in  this  category  should  still  be  effective, 
it  should  meet  the  needs  of  some  target  audience  (perhaps  not  the  target 
audience  specified)  and  it  should  support  some  of  the  goals  of  the 
curriculum.  The  range  of  potential  use  of  the  courseware  may  be  limited, 
but  what  it  does  cover,  it  should  cover  effectively. 
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VI.  GLOSSARY  OF  TERMS 

In  the  fields  of  microcomputers  and  education,  terminology  can  be 
misinterpreted.  So  that  everyone  using  the  Guide  will  interpret  words  in  the 
same  way,  a brief  glossary  is  provided  below.  Each  definition  conforms  to 
some  standard  use  of  the  term  but  is  not  necessarily  a composite  of  all  uses 
nor  the  most  common  use. 


Content 

Facts,  terms,  ideas,  concepts,  principles,  theories  and 
constructs  which  make  up  the  subject  matter  of  an 
instructional  package. 

Courseware 

Software  and  printed  material  which  support  instruction 
in  a complete  course  of  study  or  a definite  subset  of  a 
course.  In  this  sense,  it  is  not  required  that  all 

instructional  activities  be  supported  by  the  package  or 
that  the  tutorial  mode  of  instruction  be  employed.  Even 
a 30  minute  student  activity  aimed  at  one  objective  can 
be  courseware  in  this  definition. 

Documentation 

Written  material  for  the  teacher  to  use  with  the 
computer  program  (also  called  a support  booklet). 

DOS 

"Disk  Operating  System'8,  an  operating  system  which 
includes  the  capability  of  controlling  and  coordinating 
the  functions  of  a disk  drive  within  a computer  system. 

Drill  & 

Practice  Mode 

A type  of  computer  program  which  acts  as  drillmaster 
by  providing  repetitive  practice  on  a skill  or  set  of  facts. 
The  mode  assumes  that  instruction  in  the  concept  skill  or 
process  has  already  been  given.  Ideally  a drill  and 
practice  program  should:  1)  provide  random  selection  of 
exercises;  2)  record  student  progress;  3)  tailor 

instructions,  exercises  and  feedback  to  the  individual; 
and  4)  offer  corrective  feedback  upon  the  input  of 
incorrect  responses. 

Games 

Programs  in  the  instructional  gaming  mode  call  upon  the 
user  to  apply  one  or  more  specific  skills  or  concepts 
within  a "game"  environment.  The  game  dimension  of 
the  environment  includes  the  conditions  of  competition 
with  self  or  others,  specific  (if  arbitrary)  rules,  the  need 
to  develop  a winning  strategy  and  the  introduction  of 
random  events  to  force  revision  of  strategies.  The 
instructional  dimension  of  the  environment  demands  that 
the  development  both  of  identifiable  subject  matter 
skills  or  concepts  and  of  general  problem  solving  skills  be 
the  goal  of  the  exercise. 

Graphics 

Images  displayed  on  a video  screen  or  printer  which  are 
generated  by  a computer  program. 
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Hardware 

Equipment,  including  computers,  disk  drives,  cassette 
players,  cables  and  monitors. 

Materials 

Books,  folders,  envelopes,  worksheets  and  similar  items. 
"User  Support  Materials”  are  items  which  support  the 
activity  of  a person  using  a computer  program. 

Microcomputer 

A computer  system,  including  peripheral  hardware  such 
as  disk  drive  and  monitor,  based  on  a microprocessor  (or 
"chip”)  and  having  a typewriter-like  keyboard. 

Package 

One  or  more  computer  programs  with  related  materials 
and  the  storage  medium. 

Problem  Solver 

A type  of  computer  program  which  processes  data  for  a 
student  defined  problem. 

Program 

A computer  program,  written  in  Basic,  Pascal,  machine 
code  or  other  computer  programming  language.  The 
routines  and  operations  which  instruct  the  computer 
constitute  a program. 

Simulation 

Programs  in  the  simulation  mode  attempt  to  represent 
key  aspects  of  some  environment  within  which  the  user 
will  experience  the  necessity  of  making  decisions  and 
will  be  informed  of  the  results  of  his  decisions  without 
experiencing  the  real  consequences  of  possible 
misjudgements.  Simulations  are  appropriate  in  situations 
where  actual  experience  is  ruled  out  due  to  extreme 
expense,  safety  consideration  or  the  time  required  for 
the  actual  experience.  Simulations  include  problem 
solving  tasks  (e.g.  diagnosis  of  illnesses,  genetic 
experiments,  testing  theoretical  models),  procedural 
tasks  (e.g.  acid  titration,  blasting,  breeding  organisms), 
or  performances  (e.g.  control  of  water  pollution). 

Software 

Computer  programs,  including  courseware,  operating 
systems  and  languages. 

Tutorial 

For  programs  in  this  mode,  the  computer  assumes  the 
total  responsibility  for  instruction.  Tutorials  are 

characterized  by  a dialogue  between  the  student  and  the 
computer  in  which  the  direction  and  the  level  of  the 
dialogue  are  shaped  by  the  student’s  input.  Ideally, 
tutorials  should  tailor  and  present  instruction  to  the 
individual,  introduce  new  concepts,  provide  exercises  and 
tests,  monitor  student  responses,  give  immediate 
reinforcement  and  corrective  feedback,  and  keep 
records. 

Some  definitions  in  this  glossary  have  been  taken  from  Evaluator’s  Guide 
published  by  MicroSIFT  and  The  Computing  Teacher. 
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